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ABSTBACT 

The Conference Evaluation by Objectives method was 
designed to be an objective way of measureing program impact upon 
participants. At a convention prograa, speakers were asked to 
summarize their presentations, in three or four participant-centered 
objectives. Participants were given a brief, easy to fill in and easy 
to tabulate, form to evaluate the ^speaker by objectives. Eesults of 
^participant evaluation were tabulated and sent immediately to the 
speaker. ' Speaker success in accomplishing identified speech 
objectives was evaluated. This evaluation included a simplified 
statistical analysis. Oses of this evaluation method, as described in 
this sample kit of forms, procedures, tabulation guidelines and 
examples, and data interpretation sample, include pinpointing: (1) 
effective sepakers, (2) effective presentations, (3) the most 
effective portion of 4ny presentation, (U) benefits to participants, 
and (5) possible benefits to aearners in institutions sending 
participants to conventions. (Author/BJG) 
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INTRODUCTION TO CONVENTION EVALUATION BY OBJECTIVES 

This publication, AN iNTRODUCT-ION TO CONVENTION EVALUATION BY 
OBJECTIVES, is intended for boLh the gener/il dnd the specinUjied 
readei*. The general render is given an easy to understand example of 
a new way to evaluate conventions. The specialized reader is given, in 
addition to the simplilied overview, a basic statistical tool with which 
to evaluate almost any type of convention. 

This means that the genera^ reader should be able co derive ^ 
simplified overview of EVALUATION BY OBJECTIVES after reading pages 1 to 18. 
In addition, the evaluation specialist should be able to envision the 
procedures, the forms, the time,t^able, the data tabulation, and the 
statistical analysis necessary by reading pages 19 to Al. 

In order to evaluate a convention by objectives, the evaluator should 
get together v/ith the program chairman in order to look over the convention 
program. Most convention programs are uniquely developed in order to 
facilitate the objectives of the associations sponsoring the conv^mlion. 

This prepares the evaluator by objectives for CHANGE 1: 

ADD speaker-written participant-centered presention objectives 
to the existing program. 

Change 1 is not dramatic, but it does set the stage for the 
orc«'^nLv:rition ancl instrun-.ontation necessary to succeed with evaluatior 
by objactivcs. Obviously, even the amateur evaluator will now ask, *^Hcw 
does oie .^Cit the sp^iakers to develop participant-centered objectives?** 
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The answer is simple. The speakers were asked in a short letter, to 
develop objectives that would tell the participant exactly what could be 
of benefit to the listener at each presentation. A sample of this letter 
appears on page 13. It is to benoted that the sample letter gives specific 
examples of objectives correlated to a specific presentation title. 

After the -speaker-written objectives have been added to the convention 
program, it is appropriate to use a brief, easy to fill in and easy to 
tabulate, evaluation form that evaluates by objectivv.s. There are many 
different variations possible here. The form used in this example is 
reproduced on page 16. \ 

It might be noticed that, in this experimental validation of evaluation 
by objectives , an evaluation has been made both by objectives and by ' 
conventional criteria. This duplication of evaluation coverage is net 
always essential in situations with limited time and personnel. 

Inimediately, the experienced evaluator sees a certain difficulty hare, 
^he evaluator begins to envision the situation where many different 
evaluations are coming from many different sources. The resulting tower 
of Babel is enough to convince anyone that such a method can't work. 

Such a possible source of confusion has been anticipated ind prevented 
by the procedures spelled out on pages 14 and 15. These pages tell exactly 
what was done to organise and simplify the evaluation procedure. 

It must be noted here that immediate feedback to the individual speaker 
was considered of paramount importanct. This immediate feedback enables the 
speaker to get ready for next time or next year. In other words, iinnediate 
knowledge of results helps prevent a bad habit from becoming an incorrigil)le habit. 
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Also, immediate knowledge of successful results helps develop good 
habits that can produce solid learner benefits. 

This prepares the evaluator by objectives for CHANGE 2: 

USE a brief, easy to fill in and easy to tabulate, evaluation 
form that evaluates by objective's. 

All of this can be summed up in CHANGE 3: 

PROVIDE immediatc^ (within 4 weeks after the convention) 
feedback to the speaker on the results of evaluations 
from participants. 

This change 3 is a direct result of the above' mentioned CH/^NGE 2. 

Now that the speaker has submi^tQd objectives, that the briei: form 
has been utilized, and that Immediate feedback has been provided to 
speakers, it is time' for CHANGE 4: 

EVALUATE the objectives aimed at as well as the participant 
estimation of success. 

EVALUATING THE OBJECTIVES AIMED AT simply means to categoxrize, count, 
and examine the speaker-written objectives in order to make sure that these 
objectives are participant-centered and observable. In addition, care can 
be taken to make sure that these objectives" are in congruence with the 
overall aim of the convention. 

Examples of this process of evaluating the objectives can be found 
on pages 24 to 33. . 

In order to EVALUATE THE PARTICIPANT ESTIHATIOH OF SUCCESS, it is 
necessary to come up v/lth a proper tally of responses that clearly provide 
discrimination between objectives achieved and objectives not achieved. 
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obviously, there are pitfalls to avoid. These easy-to-co.Tmiit errors 
are explained on pages J4 to 38. An accurate and simple way to tabulate 
evaluations by objectives is presented on pages 39 to 41. 

Any reader who is scared off by the words *'s tatistics'* cr "tabulations" 
should look at the simplicity of the data presented on page 40 in order to 
regain courage. Anytime the data gets more complicated than that on 
page 40, both the evaluator and the decision maker should try to find out 
what errors enumerated on pages 34 to 38 have got the tabulator off the 
track. 

Some individuals instinctively feel that the long way is the best 
way. Other individuals feel that the more difficult solution is the best 
way for a disciplined mind. 

The data summarized on pages 19 to 23 points up at least two examples 
wherein long counts and detailed analyses vjere misleading because the counts 
were based upon the rules of mathematics unadapted to the particular 
requirements of a specific convention. Grouping should have come before counting 

This long winded last paragraph boils do\m to the following: sometimes 
a short, well organized counting is better than a long and involved arithmetic 
process as far as evaluation is concerned. 
This leads us to CHANGE 5: 

SIMPLIFY the math. 



CONCLUSIONS 



In order to capsulize the unique characteristics of this approach to- 
CONVENTION EVALUATION BY OBJECTIVES, the following five changes are 
presented as a concluding summary* Here are the five changes leccssitated 

CHANGE 1: 

ADD speakcr-vnritten participant-centered presentation 
objectives to the existing program. 

* 

CHANGE 2: 

USE a brief, easy to fill in and easy to tabulate, 
evaluation form that evaluates by objectives. 

CHANGE 3: ' 

PROVIDE immediate (within 4 weeks after ^the convention) 
feedback to the speaker on the results of evaluations 
from participants. 

CHANGE 4: 

EVALUATE the objectives aimed at as well as the 
participant estimation of success. 

CHANGE 5: 

SIMPLIFY the math. 



Each of these changes is illustrated in the following pages. 



1 

. t 



1 



NYSECA CONVENTION EVALUATION SUMMARY 

The purpose of the 1974 NYSECA convention evaluation was to provide 
an easy to use format upon which conference program decisions could bo 
based. This objective breaks down into two subob jectives : Subobjcctive 1 
is the PRESENTATION OF A FORMAT AND SAMPLE FORMS. This material is found 
in section I beginning on page n. 

Subobjective 2 is an ANALYSIS OF TiEE EVALUATION FORMS together with 
INDICATIONS OF SAMPLE PROGRAMMING SUGGESTIONS. This is found in 
section II which begins on page 19. 

\ 

\ 

The forms provided in section I are as follows: 

/ 

The form for evaluating INDIVIDUAL PRESENTATIONS is found 
on page 16. 

The form for evaluating both THE OVERALL CONFERENCE 
and THE FACILITIES AND SERVICE is found on page 18. 

The session by session evaluations were conducted during and 
immediately after each presentation. Detailed instructions for this session 
by session evaluation begin on page 15 « 

The overall conference evaluation and the facilities and service.^ 
evaluation was done at the evening meal preceding the final last day\of the 
convention* Specific directions for this evaluation begin on page 17. 
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Section II which presents a session-by-session evaluation is suirmiarized 
by Che following lists of correlated FINDINGS and RECOMMENDATIONS o 

FINDINGS: Out of a possible 780 evaluation forms, 563 were filled in, 

^ 

returned, and tabulated. 

RECOMMENDATIONS: The forms to be used in the 1975 evaluation should 
be as easy to fill in as the forms used in the 1974 evaluation^ 

DATA SOURCE: Page 19. 

FINDING: As easy as the forms were to fill in, the tabulation took^ 
a large amount of time since it was difficult for che tab clerks to place 
the forms alongside one another for a quick eye count. 

RECOMMENDATIONS: The response portion of the evaluation questionnaires 
should be redesigned with a view to increase tabulation speeds For example, 
the ratings such as POOR, AVERAGE, VERY GOOD, EXCELLENT, should be pushed 
more to one side to facilitate speed of tabulation. 

DATA SOURCE: This is based upon the fact that it v;as easier to 
tabulate the YES, NO, NA responses than the POOR, AVERAGE, VERY GOOD, 
•EXCELLENT responses. This is due to th6 fact that the YES, NO, NA responses 
were bunched together towards the right hand side. Refer to page 15 and 
to page 19 for examples. 
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FINDINGS: Only 29 ovaluators out of 563 completed forms made any 
comments. 

RECOMMENDATIONS: Thought might be given to devising a questionnaire 
that would be more effective in eliciting a greater percentage of response 
comments from evaluators, 
^ PATA SOURCE: Page 19. 

FINDINGS: The overwhelming majority on a 1% level of confidence 
rated all conference room evaluation items either excellent or satisfactory. 

RECOMi-IENDATIONS: Thej questions about conference rooms might be deleted 
or changed in format in order to come up with evaluation findings that can 
be used to improve the pre/gram for 1975 « 

DATA SOURCE: Page 20c 

FINDINGS: General questions used to evaluate speakers and presenters 
were answered favorably by an overwhelming majority at the 1% level of , 
confidence. This confirms statistically the impression that the vast 
majority of evaluators responded to these questions with yes. 

RECOMMENDATIONS: Either these questions be deleted since they do not 
discriminate between effective and ineffective speakers or that the yes/nn 
categorization be spread out to allow greater differentiation. 

DATA SOURCE: Pages 21-23. 
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FINDING: Seventy-eight program sessions were given, of which 6 had no 
stated objectives and 72 had stated objectives • This means that only 8% 
had no stated objectives in the convention program. 

FINDING: Of the 72 program . sessions evaluated, only one session had 
no objectives upon which to be evaluated • This means that 5 speakers were 
able to AD LIB objectives at the opening of the session. Unfortunately, 
these AD LIB objectives ^ere not recorded for inclusion in the evaluation. 

RECOMMENDATIONS: Whenever a speaker ad libs, changes, deletes, or 
adds to the objectives printed in the program, this should be recorded- by 
the room chairman. 

DATA SOURCE: Page 31 and pages 24-26. 

FINDING: It seems obvious that evaluation by objectives was accepted 
by the individual speakers at each session* 

RECOMMENDATIONS: The 1975 evaluation be conducted in the same format, 
that is, evaluation by objectives*' 

DATA SOURCE: Pages 24-31. 

FINDING: Of the verbs used for session objectives, 

44 verbs were knowledge objectives (cognitive domain), 

16 verbs were performance objectives (psychomotor domain), 

14 verbs were attitude objectives (affective domain). 
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RECOMMENDATIONS: Each speaker f orJ:he ,1 97r) convention shouh' be urged 
to iise at least one performance objective and at: least one attitude 
objective. 

DATA SOURCE: Pages 27-29. 

1 FINDINGS: Sixty-five sessions had participaTTt'Ccntered behaViO'l -1 
* objectives that were observable* Seven sessions had cither i.on-obset'yablc 
or speaker-centered objectives or botho This means tliat 10% of the 
ses«s,ions had objectives that were not up to quality control standards^ 

RECOM>ni:NT)ATIO^:S: Before the progrun/ is printed, an analysis should 
be made to ascertain that all session, objectives are both pc^vticiprnt- 
centered and . observable. Objectives that do not meet tliese criteria should 
be sent back to the speaker with suggestions for improvement, | 
DATA SOURCE: Pages 30-31. 

FINDINGS: The verb "recogni2e** was used 34 times in various session 
objectives. The verb ^'idenuify*' was used 9 times. The verb "recognize" 
was used in the sample letter ^ent to participants.. Refer to o^i,^ 33. 

RECOMt'tENDATIONS : The snmple letter sent to speakers should speciTy 
appropriate verbs for N>:SECA objectives o Similarly, the letter benu to 
.participants should specify inappropriate ve>:o<^ for objectives, Tl»is 
letter might also mention that any session whcse objectives vould be 
strictly cognitivt? should be carefully reconsidered before inclusion in 
a media type convention v;hich is interested in performance and atcitude 
objectives as well as cognitive objectives. 
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DATA SOURCE: Pages 32-3^ and page 13. 

♦ 

^ . FINDINGS: Ungroupfed tabluation of evaluation questionnaires reveals 
subtotals and totals that are misleading when used to evaluate session 
effectiveness in general or in individual ' '\c^.^. 

RECOMMENDATIONS: The tabulators of the 1975 evaluation should be 
alerted^to this danger as pointed out in the examples found on pages 34-35, 

DATA SOURCE: Pages 34-38. 

FINDINGS: At the 1% level of confidence, 



Presentations with 2 or 3 objectives produced 
the highest ranking evaluations. ' 

Presentations with 1 objective produced the 
lowest ranking evaluations. 

Presentations with 4 objectives produced the 
largest percentage of no response from 
evaluators 



RECOMMENDATIONS: - . 

1. Session speakers should be made aware of the 
fact that empirical results reveal most 
evaluators can handle two or three objectives 
in a 45 minute convention session. 

2. Speakers should be alerted to the fact that 
presentations with 4 objectives tend to 
produce a 25% no response reaction from ; 
ev^lua^tors. This could possibly be interpreted 

that 4 objectives are too many for a 45 minute session* 

3. Speakers should^ be alerted to the fact that 
presentations with 1 prestated objective tend to produce 
the largest ^percentage of average responses from evaluators. 
Since an average response to this instrument is in the 
inferior category, -it might be liypothe3i2ed that one 
objective is too global and nonspecific for- ^^^^45 minute 
conference session. * ""^--^.^ 
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DATA SOURCE: Pages -39-41, 
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FINDINGS: * In^ order to break the above tie found between presentations 
with 2 and 3 objectives at the 1% lovel of confidence, a 5% level of 
confidence analysis was used. At the 57c level of confidence, presentations 
with 3 objectives are found to bfi superior -^o presentations with 2 objectives. 

RECOxW.NDAIIOHS : Speakers :>hou:,d be made aware of tltis rinding, /oth 
as regards the suggested nuT.ber of .objectives and as regards the lev^l of 
confidence, which is 5% open to stat:Lstical chance error. 

DATA SOURCE: Page 41. 

/ 

SUMMARY .FINDINGS: As is obvious from reading the above findings, 
several overall surnmary findings stand oiit: 

1. The criteria used to select and prepare 

J the conference rooms and convention sites 

resulted in evaluations that were overwhelmingly 
positive by the vast majority at the 1% level 
of confidence. 

2, The criteria used to select session speakers 
resulted in evaluations, both on general criteria 
and on .evaluation by objective criteria, that 
resulted in overwhelmingly favorable evaluations 
at the 1% level of confidence. 

RECOMMENT)ATIONS : Since these criteria both for convention rooms to * 

lecture and preparation and for speaker selection are not currently 

/ 

available in print, the program chairmen are asked to write down these criteria 
in order to provide a record for the 1975 convention and future conventions. 

DATA SOURCE: Pages 20, 23, and 40. 
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MISCEtLANOUS FINDINGS: In addition to the evaluations received on 
official evaluation questionnaires, the following miscellanous evaluations 



1. No provision was made for estimating the 
attendance at individual program sessions. 

2. Some isolated comments are very critical 
and harsh to the speaker. / 

3. Individuals have made verbal inquiries 
about separate rooms for non-smoKers. 

4. Detailed analysis of the session-bysession 
evaluations required a long period of time. 

^ RECOMMENDATIONS: In response to the above findings, the following 
recommendations are made: 



1. On the envelope to be filled out by the 

individual room evaluation chairman, space should 
be provided on which to estimate attendance. 
For a even more detailed analysis, the 
attendance might be taken at 15 minute 

intervals such as the opening of the session, 15 minutes later, 

the second 15 minutes, and the end of the 

session. It might be interesting to notice 
such patterns as more people at a session at 
the end of the session as well as a pattern 
of less people at a session at ^ the end of a 



2k Derogatory statements should be edited out and 
not sent to individual speakers. In some cases, 
the comments should be completely deleted. At 
other times, the comments should be softened to 
correct and polite English, at least. / 

3. Convention chairman should be aware of the 

concern for separate facilities for non-smokers. 
Other , conventions sometimes urge non-smokers to sit 
on the right side of any meeting room or in a specified 
location in order to'avoid cross-ventilation by smoke* 




were^gathored from a number of other sources: 



session. 
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4. Immediately after the convention, the overall 
convention evaluations should be totaled. 

The next priority would be to tally and total 
the evaluation comments for each individual 
sess'ion. These could be immediately sent tc 
the evaluator within six weeks after the 
convention. 

The next priority would be to come up wi'ch a 
detailed analysis of the session-by-session 
results within three months after the convention. 



FINDINGS: The preceding evaluation summary analysis has been 
given in great detail with very specific explanations. This has been an 
attempt to provide a guide that can be followed by a non-specialized 
evaluator. 

RECOMMENDATIONS: The 1975 evaluation report should be simplified 
to a listing of findings, recommendations, and data sources. The data 
sources should be limited to appropriate tables numbered ai*d cross-referenced 
in thQ data sources. 

The actual tabulation and analysis should be handled by a special 
subcommittee of the NYSECA convention chairman. 

Obviously, this iipplies the recommendation, that the same format of 

analysis be used for the 1975 evaluation. ; 

^ / 

DATA SOURCE: Opinion of the BOER evaluator who ^veloped this 

/ 

format for evaluation by objective^. 
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SECTION I 
CONFERENCE EVALUATION 



Conference evaluation normally involves three different areas of 
concern: 

1. Session by session evaluation 

2. Overall conference evaluation 

3. Facilities and services evaluation 

The approach to conference evaluation herein stressed is intended 
to organize and simplify the evaluation process. This organization and 
simplication will center around a few small changes in the conference 
program that result in increased evaluation potential. 
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SESSION BY SESSION EVALUATION 

Session by session evaluation is under the aegis of the room chairman 
who handles -the explanation, distribution, and collection of evaluation 
scales. 

A set of common evaluation forms is provided to the room chairman. 

\ 

The room chairman explains to each participant the objectives for the 
session under evaluation. 

A maximum of four objectives is listed in the program un^^r the 
title and summary provided by the author. These objectives are stated 
in active verbs, the subject of which is the participant. The objectives 
for each session specify the competency or competencies to be acquired 
by participants as a result o£ listening to the presentor. The objectives 
tell what the participants will achieve as a result of this session. The 
Objectives to be evaluated do not describe what the presentor does. 

At the end of these sessions, each participant or a selected sample 
of participants receive the common evaluation forms. Each evaluator 
writes '.n the session number. These evaluation forms are presented in 
the form of a scale. The evaluators rate the achievement of all session 
objectives by using the scales provided on the common evaluation form. 
The room chairman collects the sessipn by session evaluation forms before 
the evaliiators leave the session. 

The following letter was used to explain this evaluation procedure to 
participants. 
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TMt UNIVCRSITY OF THCT bTATE OF NEW YORK 
THfl STATE EDUCATION DEPARTMENT 
ALBANY, NEW YORK 12224 



ASSOCiATL COMMlSSlO».C« f OH 
AC^CAACh ASO t.ALU*T'ON 



DIVISION OF RESEARCH 
BUREAU OF OCCUPATIONAL EDUCATION RESEARCH 
St0 474.6386 



Your presentation for the New York State Educational Communicatior Convo- 
cation in November has been received. To assist us in .preparing an evaluation 
procedure for the conference, we ask your help by writing objectives for your 
presentation. These objectives will be printed in the; final program along 
with the abstract of your presentation. 

To help you in preparing the objectives, here is an example of the format 
we would like to use: 

Title: The Director of Educational Communications 

Presented by: Kay Bailey 

Objectives: !• Recognize the areas of responsibility of a 

director in the educational programs. 
2. Evaluate your position using the guidelines presented. 



Note: , Prepare one to four objectives. 

, In writing objectives, the subject of the verb 

Is the participant. 
. The objectives specify the competencies the 

participant will take away from the presentation. 
* The objective Is not what the presentor will do, 

but what the participant will gain from what you do. 

The Program Committee meets the first weekend in June. You will then 
receive the time and room assignment for your program. Please return the infor- 
mation requested as soon as possible to: 



Kay Bailoy 

12 Michigan Avenue 

Troy, New York 12180 



Edv;ard Moy an 
Catherine M. Bailey 
n9p" Progrg^hairraen 



Sincerely yours. 
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EVALUATION DIRECTIONS 
FOR ROOM CHAIRMAN 

Each room chairman will receive evaluation forms for 10 
attendees at each session and one evaluation form for the room 
* ' chairman* 

Before the presenter starts at. each session, the room chairman 
will read to all participants the objectives upon which the session 

will be evaluated* 

The room chairman will collect the evaluation forms following 
each session and return the evaluation forms at the completion of 
■his duties in the envelope provided to J.V. Cowan at the registration 
• desk. 
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CONFERENCE EVALUATION 



OUTLINE OF PROCEDURES FOR PHASE I 



i 

. A« Each room chairman receives an envelope with 10 evaluation forms for 
each session© 



B. Each room chairman distributes 10 evaluation forms to 10 participants* 

C. Each room chairman collects all evaluation forms from the participant 
evaluators at the end of the sessibno i 



D. The individual evaluation forms for each session are kept together* 



E. All individual evaluation forms are turned into the conference 
evaluation comraitteeo 



A total tally is made for each evaluation item for each session* 



G. The comments for each session are typed up* 
For each session, the following is prepared: 

1©^ The speaker's name and address 

2. A presentation summary specifying 

(a) date 

(b) time ^ 

(c) room 

(d) title 

(e) presentors 

(f) presentation summary 

(g) participant objectives 

3. A summary of comments made by participating evaluators 
When no comments have been made, notation will be made 
of this* 

4. A statistical summary of evaluation reactions of each 
item of the evaluation form 



!• Two photocopies will be made of the results of phase I as outlined 
above* 

One copy Js for the presentor to be sent along with 
the "Thank You" letter. 

The other copy is for the files of the conference 
committee* 

ERIC 
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EVALUATION FOP^l ; 
INDIVIDUAL PRESENTATION 



DIRECTIONS: 1. Please fill in the following: 
Date ^ . 



Meeting room 



Title of presentation 



Presenter's name 



2. Evaluate how well each program objective was achieved by 
circling the rating. (Comments may be added.) 



OBJECriVE 1 
OBJECTIVE 2 
OBJECTIVE 3 
OBJECTIVE 4 



Poor 
Poor 
Poor 
Pool 



Average 
Average 
Average 
Average 



Very Good 
Very Good 
Very Good 
Very Good 



Excellent 
Excellent 
Excellent 
Excellent 

of the following. 



3. Circle YES, NO, or NA (not applicable) for each 
(Comments may be added.) 

A. Topic was of practical value 
'B. Audience interest and reaction was positive 

C. Presentation was well organized 

D. Use of media was well integrated 

E. Program schedule was maintained 

F. Session offered stimulating u^r new material 

G. Audience questions were well answered 

H. Meeting room' was adequate ^ ' 

I. Meeting room should be used again next year 

4* Please evaluate the following by checking under excellent, 
satisfactory, or unsatisfactory. Leave an item blank where 
not applicable. 



YES 


NO 


NA 


YES 


NO 


nA 


YES 


'NO 


NA 


YES 




NA 


YES 


NO 


NA 


YES 


-NO . 


NA 


YES 


NO. 


NA 


YES 


NO 


NA 


YES 


Hp 


NA 



EXCELLENT 



SATISFACTORY 



UNSATISFACTORY 



CONFERENCE ROOM 



Seating capacity 
Temperature control 
Ventilation 
Sound 

Room layout 
Lighting 

Other: 
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■ \ . • : 

OVERALL CONFEIUSNCE EVALUATION 
and 

FACILITIES AND SERVICES EVALUATION 

The overall conference evaluation and the facilities and services 
evaluation is included in the conference program. The place might be 
the last page or pages that could be removed from the program. 

Evaluation directions are included with these two evaluation forms. 
It might well be that side one would include overall conference evaluation 
while side two would include facilities and services evaluation. 

There are a number of ways to collect this material. Each 
participant might return the evaluation forms before leaving the 
conference. It might be possible to mail in the evaluation forms with a 
self-addressed return envelope. 

Placing these evaluation forms in the program itself gives each 
evaluator a chance to reflect thoughtfully before filling out the form. 
Similarily, each evaluator will be able to choose the time and place 
most appropriate for a thorough evaluation. 

NOTE: Because some participants are present for only one day or part of a, 
day at the conference, remi.hders should appear on eyei7 program page 
to fill in the overall and^ facilities evaluations. 

In order to facilitate return through the mail, it is suggested that 
a_ pre-addressed businessZreply format be employed^ 

/ 

If the evaluation form is on the last page or two pages, it is possible 
to fold, staple and place the evaluation in the mail# 
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EVALUATION FORM : 

ovEi^LL coni\h:rence 

and 

FACILITIES AND SERVICE 

\ 

DIRECTIONS: Please answer the following questions: ^ 
OBJECTIVES: What did you expect to get out of this -conference? 



Has the conference lived up to your expectations? YES 
^Valuations: Wh?,t did you get out of this conference? 



RESOURCES: For you, personally, what was the best part of this conference 



Hov; do you rate the following? 


(Circle 


your rating,) 




General sessions: 


Poor 


Average 


Very 


Good 


Excellent 


Program; 


Poor 


Average 


Very 


Good 


Excellent 


Exhibits: 


Poor 


Average 


Very 


Good 


Excellent 


Exchange of ideas: 


Poor 


Average 


Very 


Good ^ 


Excellent 


DINING FACILITIES 


Poor 


Average 


Very 


Good 


Excel lent 


Food 


Poor 


Average 


Very 


Good 


Excellent 


Seating 


Poor 


Average 


Very 


Good 


Excellent 


Service 


Poor 


Average 


Very 


Good 


Excellent 


Other: 


Poor 


Average 


Very 


Good 


Excellent 


LODGING ROOMS 


Poor 


Average 


Very 


Good 


Excellent 


Cleanliness 


Poor 


Average 


Very 


Good 


Excellent 


Housekeeping 


Poor 


Average 


Very 


Good 


Excellent 


Temperature Control Poor 


Average 


Very 


Good 


Excellent 


Beds 


Poor 


Average 


Very 


Good 


Excellent 


Sound 


Poor 


Average 


Very 


Good 


Excel lent 


Light Control 


Poor 


Average 


Very 


Good 


Excellent 


Lavatory 


Poor 


Average 


Very 


Good 


Excellent 


Other: 


Poor 


Average 


Very 


Good 


Excellent 
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SESSION BY SESSION EVALUATION 

CONFERENCE ROOMS \ 

The chart on the following page, COUNT TABUT^TION OF EVALUATION 
QUESTIONS ON CONFERENCE ROOMS, suntmarizes 563 questionnaires • This chart 
^summarizes the conference room evaluation section of the session-by-session 
evaluation form* 

Analysis has been made of this data by lumping together excellent and 
satisfactory and comparing this with unsatisfactory. After statistical 
analysis by formulas used to test the significance of a difference, both in 
a count and in a percentage, no statistically significant difference has , 
been detected among the data. at either the five percent or the ten percent 
level of co\-tf Ldence. 

-Since the small count and small percentage finding the conference 
rooms unsatisfactory is not statistically significant, it can be confidently 
said that the vast majority of conference attendees found the conference 
rooms satisfactory or excellent. 

For next year, a suggestion might be made: since a large amount of 
tabulation was necessary to come up with this statistically insignificant 
result, the questions about conference rooms might either be deleted or 
changed in format in order to come up with evaluation findings that, can 
be used to improve the program for the next year. 

A case in point is the large number of NR, or no response, under other. 
It was felt that this would encourage many participants to add significant 
comments. As a matter of fact, only 29 evaluators out of a possible 563 ^^^^ 
any comment. This might point up the need to devise a questionnaire that would 
be more effective in eliciting a greater percentage of response from evaluators 

2B. 



Count Tabulation of 
Evaluation Questions on 
Conference Rooms 
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Excellent 


Satisfactory 


Unsatisfactory 


Subtotal 


NR 


Tota 


Seating 


307 


202 


29 


538 


25 


563 


Temperature 
Control 


220 


256 


66 


542 


21 


563 


Ventilation 


230 


255 


A5 


530 


33 


563 


Sound 


209 


218 


75 


502 


61 


563 


Room Layout 


205 


273 


50 


528 


35 


• 563 


Light 


230 


270 


16 


516 


47 


563 


Other 


9 


4 


16 


29 


534 


563 
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SESSION BY SESSION EVALUATION 

The chart on the following page, COUNT TABULATION OF EVALUATION 
QUESTIONS A THROUGH I, summarizes the count of responses to evaluation 
questions A through I, 

These evaluation questions give the evaluator a chance to respond 

/ 

with yes, no, or not applicable to questions such as: 

Topic was of practical interest 

Audience interest and. reaction was positive 

Presentation was well organized 

Use of media was well integrated 

Program schedule was maintained 

Session offered stimulating or- new material 

Audience questions were well answered ' 

Meeting room was adequate 

Meeting room should be used again next year 

As is obvious from a casual glance at these questions, these evaluation 
items reflect typical concerns found on most conference evaluation forms. 
From this point of view, questions A through I were well designed* 

On the other hand, questions A through I do not appear to be very 
discriminating when compared with the results of an objective-by-objective 
evaluation of individual speakers. 




For example, some speakers and presentors who were evaluated poor by-~^ 
the majority of evaluators were evaluated yea on questions A through I by 
a majority of evaluators. 

A casual glance at the count tabulation for questions A through I 
reveals Lhe fact that almost all of the evaluators responded with yes to 
almost: all of the questions* This raises a question as to how valuable 
those questions are as discriminating items. 

Statistical analyro'? performed on this data both as a count and as a 
percentage analysis reveals that all of these questions were statistically 
significant at the VL level of confidence. This confirms statistically the 
impression that the vast majority of evaluators responded to these questions 



In fact, the only place where Lhe analysis was not statistically 

I 

significant for all the items appearing in the ITR or no response column. 
For question 3G (audience questions were veil answered), the NR response 
was statistically significant at the 5% level of confidence. This simply 
means that for some reason or other a statistically significant number of 
people did not respond to this question. The other no response questions 
were what might be expected by chance. 

This again raises the question, ''Why use such questions A through I 
when the responses are going to be one sided and when the responses did not* 
discriminate poor speakers from excellent speakers?" 

As will be explained in the next section, discrimination was possible 
between poor speakers and excellent speakers on the basis of an evaluation j 
by objectives. This conclusion might inply that more significant evaluation 
results would be obtained by focusing on objectives rather than on speaker 
and presenter variables represented by questions A through I. 




with yes- 



ERIC 



29 



I- 



Count Tabulation of 
Evaluation Questions A through I 





Yes 


No 


Not 
Applicable 


Subtotal 


NR 


Total 


3A 


525 


13 


14 


552 


11 


563 


JO 


/i"7 A 


J J 


13 


520 


43 


563 


3C 


481 


45 


3 


529 


44 


563 


3D 


449 


51 


24 


524 


39 


563 


3E 


451 


53 


1 


505 


58 


563 


3F 


442 


62 


8 


512 


51 


563 


3G 


440 


16 


'28 


484 


79 


563 


3H 


462 


66 


3 


531 


32 


563 


31 


420 


73 


16 


509 


54 


563 
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SESSION BY SESSION EVALUATION 
EVALUATION BY OBJECTIVES 

ANSWERS TO TflE QUESTION, '^HW MANY OBJECTIVES DID DIFFERENT PRESKNTORS USE?" 

In answer to the question, "How many objectives did different presenters 
use?" the chart on the following page, COUNT AND PERCENTAGE ANALYSIS OF TYPES 
OF PRESENTATIONS CLASSIFIED BY NUMBER OF OBJECTIVES, is presented. 

In brief, according to the 78 speakers and presentations evaluated, 36% 
used three objectives, 29% used two objectives, 21% used four objectives, 13% 
used one objective, and 1% used zero objectives. 

This means that most presentors used 3 or 2 objectives. 

It is interesting to note, that as will be pointed out in the following 
section, only one session and presentation, that is, only one presenter, 
did not have objectives upon which to be evaluated. In the original program, 
six sessions, that is, six presentors, had no stated objectives. Tliis means 
that of the six presentors who submitted no objectives for the program, five 
developed, "ad lib" objectives at the openning of the session. 

This phenomenon is interpeted as an^acceptance of objectives on the part 
of speakers. 
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One of the following charts, ALPHABETICAL LIST OF VERBS USED IN 
NYSECA OBJECTIVES, presents all of the verbs used. These verbs are 
subdivided into three categories: knowledge objectives, performance 
objective^, and attitude objectives* 

Statistically, 44 verbs were knowledge objectives; 16 verbs were 
perforinance objectives; and 14 verbs were attitude objectives. 



Count and Percentage Analysis of 
Types of Presentations Classified 
by Number of Objectives 



Presentations with 
4 objectives 



Count of 
How Many 



16 



Percentage 
(Based on Total of 
78 Presentations) 



21% 



Presentations with 
3 objectives 



28 



36 



Presentations with 
2 objectives 



23 



29 



Presentations with 
1 objective 



10 



13 



Presentations with 
1 objective 



Total = 78 
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Alphabetical List of 
Verbs Used in NYSECA Obiectives 




K 


P 


A 


Acquaint administrators 


Acquire techniques 


Accept 


Acquire inforination 


' Assist 


Appreciate 


A nn 1 V 


Communicate 


Awake 


Bg(coi;ig) aware 


Develop procedures 


Be(come) interested 


Be coerizant 


Emulate 


Desire 


Classif V 


Enrich 


Encourage 


Compare 


Exhibit 


Establish 


Correl ate 


Experience 


Evaluate 


Define 


Gain practical knowledge 


Feel 


Dp <;f*T"l he 


Give examples (model) 


Increase involvement 


Design 


Improve 


Motivate 


Designate 


Make 


Promote 


Dp tp rminp 


Offer opportunity 


Rate 


Dl ^icnvPT' 

W V C ^ 


Train 


Select 


Discuss 


Utilize 




Examine 


Visit 




Explore 




- 


Formulate 






Gain a conception 






Gather ide:is 






Heighten consciousness 






Identify 






Know 






Learn 






Make aware 


o4 





Name (list) 

Observe 

Organize 

Perceive 

Plan 

Present information 
Promote insight 
Realize 

Receive suggestions 

Recognize 

Review 

See ' 

Show 

State 

Survey 

Trace 

Understand 

Update 

Visualize 
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DOMAIN ANALYSIS 

AA verbs are knowledge objectives (cognitive domain) 

16 verbs are performance objectives (psychomotor domain) 

lA verbs are attitude objectiv2s (affective domain) 
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ANSWERS TO THE QUESTION, *MlOW COOD WERE THE 
SPEAKER-DEVELOPED .PRESENTATION OBJECTIVES?" 

I 

I 

In answer to the question, *^How good were the speaker-developed 
presentation objectives?", the chart of the following page, AN ANALYSIS OF 
NYSECA PROGRAM OBJECTIVES BY CONTENT, is presented. 

In briefs several questions were posed to analyze the quality of the 
speaker-developed presentation objectives. Each individual objective was 
analyzed by asking the following questions: 

Is this objectives SPEAKER-CENTERED or 
PARTICIPANT-CENTERED ? 

Is the activity of this objective 

OBSERVABLE or NON-OBSERVABLE? ^ 

Is this objective PRESPECIFIED in the 
convention program or AD LIB sometime 
during the presentation? 

Those objectives which were rated as PARTICIPANT-CENTERED, OBSERVABLE, 
and PRESPECIFIED were given the highest quality evaluation. 

There are obviously many other quality controls for objectives that 
can be introduced* In order to avoid contamination by subjective opinion, 
it is felt necessary to be objective here rather than subjective. 
Objective here means stating the exact word or words used by the speakers 
in order to allow readers of this evaluation report to judge the quality 
and appropriateness of each individual verb for each individual presentation. 
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An Analysis of NYSECA Program Objectives 
by Content 

Program Objectives : 

78 program sessions were given, of which, 

6 (8%) had no stated objectives and_ 
72 (92%) had stated objectives 

^) Quality of Objectives ; 

Of the 72 program sessions with objectives 

65 sessions (90%) had participant-centered behavirral 
objectives and 

7 sessions (10%) had either non-observable or speaker 
centered objectives, or both 

Session Objectives ; 

Of the 6 presenters who submitted no objectives for the 
program, 5 developed "ad lib" objectives at the opening 
of the session* 



Evaluation Objectives : 



This means that 

77 sessions (99%) had objectives upon which to be 
evaluated and 

1 session (1%) had no objectives upon which to be 
evaluated* 



ANSWERS TO THE QUESTION, ^VllICH VERBS WERE 
USED MOST FREQUENTLY IN NYSECA SPEAKER-DEVELOPED OBJECTIVES?" 

The answer to this question is suiranarized on the following page in the 
chart, VERB FREQUENCY ANALYSIS XI^' NYSECA OBJECTIVES* 

In brief, the verb *Vecognize" was \ised 34 timeso The verb identify 
was used 14 times. The verb 'Acquire information *was used 9 times* The 
two verbs '^become aware and know* were used 8 times* It should be noted 
that the verb*'recognize"was used in the letter sent to speakers by the 
program chairman. Perhaps, several speakers took this as a quasi 
official endorsement* 

The implication here would be that the NYSECA program chairman 
should come up with a list of observable, participant-centered verbs that 
can be used by potential speakers in develc;/ng objectives for each 
presentation. 

Such a list of suggested verbs will not hamstring original thinkers. 
However, such a list will help out speakers who are not quite certain what^ 
is meant by an acceptable participant-centered objective. 

Similarly, it might be helpful for the NYSECA program chairman to 
specify a list of verbs that are not considered acceptable. 
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VERB FREQUENCY' ANALYSIS 
in NYSECA Objectives 



Times Use d Number of Verbs ' ' Verbs 

' 34 ^ 1 R'icognize 

14 , 1 ^ Identify 

9 2 ^ , Acquire information 

Ad lib 

8 2 Be (come) aware 

Knov; 

7 ' 2 Exarainje 

See i 

\ 

6 1 Utilize 

5 2 Develop procedure 

Understand 

4 3 Evaluate 

Organize 

Pre sent*, i nf ormation 

3 3 Acquire techniques 

Explore 
Learn 

2 13 Apply 

Appreciate 
Classify 
Define 
Describe 
Discover 
Discuss 
Establish 
Experience 
Gather ideas 
Improve 
Observe 
Review 

1 ' 44 44 remaining verbs 

Total 74 verbs 
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SESSION BY SESSION EVALUATION 
A STATISTICAL TABULATION DANGER TO AVOID 

Look at tho following chart, OVERALL TABULATION COUNT OF nVALliVnONS 
FOR INDIVIDUAL OBJECTIVES. 

This chart is easy to read and understand. For each questionnaire.* 
the responses have been counted for objectives 1, 2, 3, and A. In 
addition, a count has been made for NR or no response. These counts have 
been totaled up into the subtotal of completed evaluation and into the 
grand total of forms submitted by evaluatorc. 

This simplified count gives che mistaken impression that very few 
presentations had four objectives. 

o 

A casual look at the next chart,' OVERALL PERCENTAGE ANALYSIS (BASED 
UPON 563 TOTAL ^FORMS) OF EVALUATIONS FOR INDIVIDUAL OBJECTIVES, gives the 
misleading impression that the fewer the objectives, the better the 
evaluations. 

In order^ to compensate for the number of no response evaluation forms 
for specific objectives, the next chart, OVERALL PERCENTAGE ANALYSIS (BASED 
UPON SUBTOTALS OF COMPLETED EVALUATION) OF EVALUATIONS OF INDIVIDUAL 
OBJECTIVES, was developed. 

This chart gives the misleading impression that all objectives from 
1 to 4 were rated equally well v;hen one compares the excellent, very good, 
average, or poor column. 

In brief, the misleading impression from this tabulation is the fact 
that more than 80% of the prs^jentations were either very good or excellent. 
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The above observations were made in order to prepare for the next 
section which will point up a discriminating method of tabulating the 
results of an evaluation by objectives. As will become obvious, evaluation 
by objectives requires a simplified but individualized approach to tabulating 
the overall results in order to avoid errors based upon false assumptions 
about group averages. \ 
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Overall Percentage Analysis 
(Based Upon 563 Total Forms) 
of Evaluations for Individual Objectives 



PERC E NTAGES PER CATEGORY 

■■■ ■ 

/ Poor Average Very Good Excel lent NR 

Objective 1 / 2% 137, 357, 367, 14% 
/ 

Objective 2 " 2 12 29 28 29 

Objective 3 2 8 19 21 50 

Objective 4 1 4 11 13 71 
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Overall Percentage Analysis 
(Based Upon Subtotals of Completed Evaluations) 
of Evaluations for Individual Objectives 



PERCENTAGES PER CATEGORY 

Poor Average Very Good Excellent 

Objective 1 2% 17% 40% 4l% 

Objective 2 3 17 4l 39 

Objective 3 3 15 39 43 

Objective 4 2 14 36 47 



45 



39 



ANSWERS TO THE QUESTION, "HCW MANY 
OBJECTIVES SHOULD A PRESENTATION HAVE?" 



Look at the following chart, COUNT PERCENTAGE ANALYSIS, CATEGORY BY 
CATEGORY OF EVALUATIONS FOR INDIVIDUAL OBJECTIVES. This chart provides a 
count per category and a percentage per category. 

The categories have been choosen as follows: 

Presentations with four objectives 
Presentations with three objectives 
Presentations with two objectives 
Presentations with one objective 

This type of a count and percentage analysis allows each individual 
speaker to compare evaluation ratings with speakers making similar 
presentations. From a statistical point of view, the most risky number 
of presentation objectives is one. If the speaker has one objective, 
there is a 40% chance of getting an average rating. An average rating 
for this conference is in the inferior category since most get much higher 
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Count 

Percentage Analysis, 
Category by Category of 
Evaluations for Individual Objectives 



COUNT PER CATEGORY 



Poor Average Very Good Excellent NR Total 



Presentations with 
4 objectives 

Presentations with 
3 objectives 

Presentations with 
2 objectives 

Presentations with 
1 objective . 



8 
8 
9 



15 



16 
28 

23 

10 



Presentations with 
4 objectives 

Presentations with 
3 objectives 

Presentations with 
2 objectives 

Presentations with 
l objective 



0% 



PERCENTAGE PER CATEGORY 



01 



13 



40 



50% 



29 



39 



10 



25% 



5A 



39 



40 



25% 100% 
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10 
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The following conclusions can best be understood after a careful 
examination of the preceding chart, COUNT PERCEOTAGE ANALYSIS, CATEGORY 
BY CATEGORY OF EVALUATIONS FOR INDIVIDUAL OBJECTIVES o 

On the one percent level of confidence, presentations with either 
two or three objectives produce the highest ranking evaluations • 
Presentations with one objective produced the lowest ranking evaluations • 
Presentations with four objectives produced the largest percentage of 
no response from evaluators. 

On the five percent level of confidence, presentation's with three 
objectives produced the highest evaluations « In other words, the 
preeminence of presentations with either three or two objectives resulted 
in a tie on the one percent level of confidence. The slight edge of 
presentations with three objectives over presentations with two objectives 
is only at the five percent level of confidence. 
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